Here is the demo if anyone is interested.

Awesome, great job!

The VSCode extension should probably be maintained as a separate repository, but that would be an easy change. If this PR works, we can add it to a repository and only merge the LPython change.

@namannimmo10 would you please have time to test this PR out?

(Do not merge, we have to do the above refactoring.)

This is looking great @ankitaS11!! From my cursory reading of this PR you are adding a second Language Server that is meant to serve the diagnostic messages for LPython. If that is the only thing you are planning on doing, might I suggest you implement this as a linter instead? Basically the only thing that you need is an interface to provideCodeActions. I have drafted a mostly complete extension doing that (can't compile lpython with the above instructions for some reason, need to investigate further, probably I am missing something in my conda env).

Basically the gist of it is, that if you want a standalone feature from LSP e.g. diagnostics, you don't have to implement the entire server, you can just implement the interface to that single feature and VS Code is capable of merging all the providers.

This is looking great @ankitaS11!! From my cursory reading of this PR you are adding a second Language Server that is meant to serve the diagnostic messages for LPython. If that is the only thing you are planning on doing, might I suggest you implement this as a linter instead? Basically the only thing that you need is an interface to provideCodeActions. I have drafted a mostly complete extension doing that (can't compile lpython with the above instructions for some reason, need to investigate further, probably I am missing something in my conda env).

Basically the gist of it is, that if you want a standalone feature from LSP e.g. diagnostics, you don't have to implement the entire server, you can just implement the interface to that single feature and VS Code is capable of merging all the providers.

Hi @gnikit

First of all, thanks for the discussion in the previous PR about using the VSCode's language server existing module, and that helped a lot. :)

Just to clarify, this will be the only language server (for LPython) once, and if this PR is merged (I'll remove the previous implementation of JSONRPC and LpythonServer in C++ in the next commits), and will consequently also add other features like document symbol lookup. Do you think that solves your question?

Hi @gnikit

First of all, thanks for the discussion in the previous PR about using the VSCode's language server existing module, and that helped a lot. :)

No worries, it's my pleasure helping out, feel free to drop me an email if you have any questions with LSP.

Just to clarify, this will be the only language server (for LPython) once, and if this PR is merged (I'll remove the previous implementation of JSONRPC and LpythonServer in C++ in the next commits), and will consequently also add other features like document symbol lookup. Do you think that solves your question?

I see. Removing the JSONRPC from C++ sounds like a good idea but if your parser (libASR) is in C++ you will still have to communicate the results from C++ to whatever language you choose to implement your LS. You can use normal JSON for that like you are doing here, or TOML, or some other custom format.

From my understanding you are currently sending your diagnostics from C++ in JSON, then parsing them into a vscode.Diagnostic[] in Typescript. If you want to to that with symbols, hover, rename or any other LSP request you will have to create your own interface that will carry all the necessary info to then be converted into a JSONRPC request in Typescript.

The underlying problem basically is that your semantics (libASR) are in C++ and there is no straightforward way to interface with Typescript. To do that you would basically have to reinvent the JSON requests already present in the LSP. Does that make a bit more sense?

I will drop a message to the Microsoft Team that produced proprietary Language Servers in C++ to see if they can disclose any details.

Right now the current code is still maintainable, so I suggest we follow our current design and get everything working.

I agree with @gnikit that there is probably a better way to do this, and if so, we should definitely refactor our code to use the better approach. I'll meet with @gnikit next week to discuss this more, and we can even meet all three of us to brainstorm the design.

We'll use exactly the same approach with LFortran also later on.

Hi, @certik and @gnikit

Let me quickly explain how we are communicating the TypeScript Language Server with C++ libASR: (https://github.com/ankitaS11/lpython/blob/lsp/vscode-ts/editor/vscode/lsp-sample/server/src/server.ts#L246)

const stdout = await runCompiler(text, "--show-errors ", settings);

What this means is, that we communicate with the compiler and not with libASR, so we rely on the compiler to have an appropriate flag that will just print the output to the console. Yes, we'll expect it to print us the necessary information in a decided structure (example: it should have line details if it's diagnostics, and with specific names to those values), which is (in my opinion) alright since we'll have to have a standard for our own communication at some stage.

This function runCompiler is implemented here, which is essentially an asynchronous function.

This means, that when we try to add support for document symbol lookup, we'll just need a flag in LPython compiler, something like: --show-symbols which will print (to stdout) a structure in the form of a string (converted from a JSON object) - and this will be caught by our server in TypeScript, parsed and sent to the appropriate callback function in our extension.

I haven't looked yet at what information is required for hover or other features, but I plan to look at them next week, just to get an idea of what structure we can have.

For LFortran, this will be the same, that we just have the same flags, and honestly speaking, I found this as the only way (to my knowledge - which might be limited at this point in time) to have a server in TS (which enables us to do more and better) communicating with the compiler.

Right now the current code is still maintainable, so I suggest we follow our current design and get everything working.

@certik - By "current code", do you mean the code in this PR or the LSP code we had merged before? Just wanted to clarify!

I'll meet with @gnikit next week to discuss this more, and we can even meet all three of us to brainstorm the design.

I think a discussion will help! I'll be happy to join and discuss or brainstorm the ideas/design.

Great work. Thanks, @ankitaS11. I tested this locally and works as expected.
To me, this looks more like a linter; so if we are going to eventually have just one language server, and we add various functionality to it, e.g., code formatting, strict type checking, etc, that act as a sort of plugin, then it would be great.